Controller for interfacing with a computing program using position, orientation, or motion

ABSTRACT

Methods and systems for interfacing a controller device with an interactive program executing at a computer are presented. A controller includes a camera for capturing images of a display. The position of the controller is determined based on the perspective distortion, size, location and orientation of the display in the captured images. The controller may additionally include an IR projector and an IR camera. The IR projector generates an IR beam, which creates a projection on a surface. The IR camera captures IR images of the projection. The distance of the controller from the surface and its relative orientation are determined based on analysis of the IR images.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to application Ser. No. ______ (AttorneyDocket No. SONYP099B), the disclosure of which is incorporated herein inits entirety.

BACKGROUND

1. Field of the Invention

The present invention relates to methods and systems for interfacing acontrol device with a computer device, and more particularly, methodsand systems for interfacing a control device with a computer programexecuting at a base computing device.

2. Description of the Related Art

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce a very realistic game experience.

Example gaming platforms, may be the Sony Playstation®, SonyPlaystation2® (PS2), and Sony Playstation3® (PS3), each of which is soldin the form of a game console. As is well known, the game console isdesigned to connect to a monitor (usually a television) and enable userinteraction through handheld controllers. The game console is designedwith specialized processing hardware, including a CPU, a graphicssynthesizer for processing intensive graphics operations, a vector unitfor performing geometry transformations, and other glue hardware,firmware, and software. The game console is further designed with anoptical disc tray for receiving game compact discs for local playthrough the game console. Online gaming is also possible, where a usercan interactively play against or with other users over the Internet. Asgame complexity continues to intrigue players, game and hardwaremanufacturers have continued to innovate to enable additionalinteractivity and computer programs.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between user and the gaming system. One way ofaccomplishing a richer interactive experience is to use wireless gamecontrollers whose movement is tracked by the gaming system in order totrack the player's movements and use these movements as inputs for thegame. Generally speaking, gesture input refers to having an electronicdevice such as a computing system, video game console, smart appliance,etc., react to some gesture made by the player and captured by theelectronic device.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present invention provide methods and systems forinterfacing a control device with a computer program executing at a basecomputing device. It should be appreciated that the present inventioncan be implemented in numerous ways, such as a process, an apparatus, asystem, a device or a method on a computer readable medium. Severalinventive embodiments of the present invention are described below.

In one embodiment, in a computer-based system having a display and acontroller device for interfacing with an interactive program, a methodfor determining the position of the controller device is provided.According to the method the computer-based system is initialized toestablish a starting position of the controller device relative to thedisplay. The method operation of initializing includes (i) capturing animage of the display using a camera integrated in the controller device;(ii) capturing an image of a radiation pattern projected by thecontroller device; and (iii) analyzing the captured image of the displayand the image of the radiation pattern projected by the controllerdevice to determine a size of the display in the captured image at adistance of the controller device as determined by the image of theprojected radiation pattern.

Further according to the method for determining position, successiveimages of the display are captured at the controller device. Then, aposition of the controller device relative to the display is determinedbased on a perspective distortion of the display in the capturedsuccessive images of the display. The determined position of thecontroller is provided to the computer-based system to interface withthe interactive program to cause an action by the interactive program.

In one embodiment, an orientation of the controller device is determinedbased on a location and orientation of the display in the captured imageof the display. In one embodiment, the orientation of the controllerdevice is selected from the group consisting of pitch, roll, and yaw.

In one embodiment, the movement of the controller device is determinedby tracking the position of the controller at the time of each capturedsuccessive image.

In one embodiment, the radiation pattern projected by the controller isgenerated by an IR projector.

In one embodiment, when the position of the controller device cannot bedetermined based on the captured successive images of the display, thenauxiliary position determination is performed. First, the radiationpattern is projected from the controller device. Then successive imagesof the radiation pattern are captured at the controller device. And theposition of the controller device is estimated based on the capturedsuccessive images of the radiation pattern.

In one embodiment, the operation of capturing an image of the displayincludes capturing an image of a portion of the display sufficient toenable determination of the size of the display.

In one embodiment, the operation of capturing successive images of thedisplay includes capturing successive images of a portion of the displaysufficient to enable determination of the position of the controllerdevice.

In accordance with another embodiment of the invention, in acomputer-based system having a display and a controller device forinterfacing with an interactive program, a method for determining theposition of the controller device is provided. First, dimensions of thedisplay input by a user of the computer-based system are received. Then,successive images of the display are captured at the controller device.A position of the controller device relative to the display isdetermined based on the dimensions of the display and a perspectivedistortion of the display in the captured successive images of thedisplay. Then, the determined position of the controller is provided tothe computer-based system to interface with the interactive program tocause an action by the interactive program.

In one embodiment, the dimensions of the display include an aspect ratioof the display and a diagonal measurement of the display.

In accordance with another embodiment of the invention, a system forinterfacing with an interactive program is provided. The system includesa display for displaying the interactive program. A controller device isprovided for interfacing with the interactive program. The controllerdevice includes a projector for projecting a radiation pattern, a firstcamera for capturing an image of the projected radiation pattern, and asecond camera for capturing images of the display. Initialization logicis provided for initializing the system to establish a starting positionof the controller device relative to the display. The initializationlogic is configured to (i) capture an image of the display using thesecond camera; (ii) capture an image of a radiation pattern projected bythe projector using the first camera; and (iii) analyze the capturedimage of the display and the image of the radiation pattern to determinea size of the display in the captured image at a distance of thecontroller device as determined by the image of the projected radiationpattern. Position determination logic is provided for determining theposition of the controller device relative to the display. The positiondetermination logic is configured to (i) capture successive images ofthe display at the controller device using the second camera, and (ii)determine a position of the controller device relative to the displaybased on a perspective distortion of the display in the capturedsuccessive images of the display. Communication logic provides thedetermined position of the controller to the computer-based system tointerface with the interactive program.

In one embodiment, orientation determination logic is provided fordetermining an orientation of the controller device based on a locationand orientation of the display in the captured successive images of thedisplay. In one embodiment, the orientation of the controller device isselected from the group consisting of pitch, roll, and yaw.

In one embodiment, movement determination logic is provided fordetermining movement of the controller device by tracking the positionof the controller at the time of each captured successive image.

In one embodiment, auxiliary position determination logic is configuredto perform the following operations when the position of the controllerdevice cannot be determined based on the captured successive images ofthe display: (1) projecting the radiation pattern from the controllerdevice; (2) capturing successive images of the radiation pattern at thecontroller device; and (3) estimating the position of the controllerdevice based on the captured successive images of the radiation pattern.

In one embodiment, the capture of an image of the display includescapture of an image of a portion of the display sufficient to enabledetermination of the size of the display.

In one embodiment, the capture of successive images of the displayincludes capture of successive images of a portion of the displaysufficient to enable determination of the position of the controllerdevice.

Other aspects of the invention will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 illustrates a generic interactive system, in accordance with anembodiment of the invention.

FIG. 2 illustrates an exemplary controller device, in accordance with anembodiment of the invention.

FIG. 3 illustrates a detailed view of the motion capture subassembly, inaccordance with an embodiment of the invention.

FIG. 4A, illustrates a controller with an IR projector emitting apatterned radiation beam towards a surface, in accordance with anembodiment of the invention.

FIG. 4B illustrates a cross-sectional view of a conically shapedpatterned radiation beam generated by a controller striking a surface,in accordance with an embodiment of the invention.

FIG. 4C illustrates a radiation beam deviated by an angle θ from anorthogonal to a surface, in accordance with an embodiment of theinvention.

FIG. 4D illustrates a radiation beam in a similar configuration as thatshown at FIG. 4C, wherein the controller has been rotated by 90 degrees,in accordance with an embodiment of the invention.

FIG. 4E illustrates a controller emitting an IR beam having a triangularpatterned cross-sectional shape.

FIG. 5 illustrates a user interface for entering the dimensions of adisplay, in accordance with an embodiment of the invention.

FIG. 6A illustrates an example of a pattern for aiding in detection ofthe outer frame of a display, in accordance with an embodiment of theinvention.

FIG. 6B illustrates an example of a pattern for aiding in detection ofthe outer frame of a display, in accordance with an embodiment of theinvention.

FIG. 7A illustrates a controller located in front of a display, suchthat the display appears in a captured image taken by an RGB camera ofthe controller, in accordance with an embodiment of the invention.

FIG. 7B illustrates a controller moved close to a display such that thedisplay appears larger in an image captured at the controller, inaccordance with an embodiment of the invention.

FIG. 7C illustrates a controller moved to the left side of a display,such that the image of the display as seen in a captured image taken atthe controller exhibits perspective distortion resulting from thelocation of the controller, in accordance with an embodiment of theinvention.

FIG. 7D illustrates an overhead view of a controller, wherein the yaw ofthe controller has been negatively shifted, in accordance with anembodiment of the invention.

FIG. 7E illustrates an overhead view of a controller, wherein the rollof the controller has been positively adjusted, in accordance with anembodiment of the invention.

FIG. 7F illustrates a side view of a controller, wherein the pitch ofthe controller has been positively shifted, in accordance with anembodiment of the invention.

FIG. 8A illustrates a display, wherein a series of tracking bars arerendered at the borders of the screen of the display, in accordance withan embodiment of the invention.

FIG. 8B illustrates a display, wherein reflective stickers are attachedto the corners, in accordance with an embodiment of the invention.

FIG. 9A illustrates a controller having multiple RGB cameras withoverlapping fields of view, in accordance with an embodiment of theinvention.

FIG. 9B illustrates a controller having multiple RGB cameras withoverlapping fields of view, in accordance with an embodiment of theinvention.

FIG. 10A illustrates a controller having an RGB camera with a fisheyelens, in accordance with an embodiment of the invention.

FIG. 10B illustrates a controller having multiple RGB cameras withfisheye lenses, in accordance with an embodiment of the invention.

FIG. 11 illustrates the functionality of a controller at variouspositions and orientations, in accordance with an embodiment of theinvention.

FIG. 12A illustrates a controller directed towards a display, inaccordance with an embodiment of the invention.

FIG. 12B illustrates a controller being turned away from a display, suchthat IR functionality of the controller is initiated, in accordance withan embodiment of the invention.

FIG. 12C illustrates a controller directed away from a display, inaccordance with an embodiment of the invention.

FIG. 13 illustrates a user holding a controller in various positions andorientations, in accordance with an embodiment of the invention.

FIG. 14 illustrates a schematic diagram of a system for providingcontroller input to an interactive program, in accordance with anembodiment of the invention.

FIG. 15 illustrates the components of a handle of a controller with anexpansion connector, in accordance with an embodiment of the invention.

FIG. 16 illustrates a controller with sensors for improving movementtracking, in accordance with an embodiment of the invention.

FIG. 17A depicts an attachment for the handle of a controller, inaccordance with an embodiment of the invention.

FIG. 17B illustrates an embodiment where the attachment of FIG. 17A isconnected to the controller of FIG. 15.

FIG. 18 illustrates hardware and user interfaces that may be used todetermine controller location, in accordance with an embodiment of thepresent invention.

FIG. 19 illustrates additional hardware that may be used to processinstructions, in accordance with an embodiment of the present invention.

FIG. 20 illustrates a server-based game-computing system supportingmultiple users simultaneously, in accordance with an embodiment of theinvention.

FIG. 21A illustrates a method for initializing a motion controller todetermine the size of a TV display, in accordance with an embodiment ofthe invention.

FIG. 21B illustrates a method for run-time tracking of a TV displayouter frame with an RGB camera, in accordance with an embodiment of theinvention.

FIG. 22 illustrates a method of using a laser distance measure tocalibrate TV display physical size, in accordance with an embodiment ofthe invention.

FIG. 23 illustrates a method for providing interaction with aninteractive program, in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

The following embodiments describe methods and apparatus for interfacinga control device with a computer program executing at a base computingdevice by using visual cues for both user feedback and input to thecomputer program.

It will be obvious, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentinvention.

Throughout this detailed description of various exemplary embodiments ofthe invention, reference will be made to analysis of images to determineobjects or the properties of objects within the images. Such analysismay employ any of various image recognition methods and techniques asare known in the art, without departing from the spirit and scope of theinvention.

FIG. 1 illustrates a generic interactive system 101. The system includesa computer 103 and a display 106. In various embodiments, the computer103 may be a general purpose computer, a special purpose computer, agaming console, or other such device which executes an interactiveprogram that is rendered on the display 106. Examples of gaming consolesas are known in the art include those manufactured by Sony, Microsoft,Nintendo, and the like. The display 106 may be a television, a monitor,a projector display, or other such displays and display systems whichare capable of receiving and rendering video output from the computer103. A user 102 provides input to the interactive program by moving acontroller 100. In a preferred embodiment, the controller 100communicates wirelessly with the computer 103, as this provides forgreater freedom of movement of the controller than a wired connection.

FIG. 2 illustrates an exemplary controller 100. The controller 100 asshown is designed to be handheld by the user 102. Various buttons 202are included for providing input to the interactive program. Theparticular functions of the buttons 202 are determined by theinteractive program, and each of the buttons 202 may or may not beconfigurable or assignable by the user 102 in accordance withspecifications of the interactive program. Trigger buttons 203 andjoystick 204 provide additional intuitive mechanisms for generating userinput. Though the controller 100 as shown is designed to be handheld, inother embodiments of the invention, the controller 100 may be designedfor manipulation by the user 102 in other ways. For example, thecontroller 100 may be attached to the user 102 by means which are knownin the art, such as a strap or harness. Or the controller 100 may bemounted to or take the form of an object which may be moved under thecontrol of the user 102. For example, the controller 100 may be part ofa steering wheel assembly, an instrument-like device, or other devicesor assemblies which may exhibit movement for the purpose of providinginput to the interactive program. Additionally, the controller 100includes a motion capture subassembly 206, which consists of varioushardware components specialized to enable determination of the positionand motion of the controller 100.

It will be understood by those skilled in the art that the use of theterms motion capture, motion controller, and the like are to be broadlyconstrued. Motion capture may include capture of position andorientation at specific points in time, as well as the capture ofchanges in position and orientation occurring over time. Position willgenerally be described with reference to a location in athree-dimensional space (e.g. as indicated by x, y, z coordinates).Orientation will generally be described in terms of pitch, yaw and roll.

FIG. 3 illustrates a detailed view of the motion capture subassembly206. An IR projector 208 is included for generating a beam of IR ornear-IR radiation 219. The IR projector 208 includes an IR LED whichemits radiation in the IR or near-IR spectrum. In one embodiment, the IRLED 210 is disposed at one end of a reflective tube 212. Radiationemitted by the IR LED is harnessed within the reflective tube 212. Atthe opposite end of the reflective tube 212, an aperture 214 is providedwhich allows radiation to escape from the reflective tube 212, so thatradiation passing through the aperture 214 is substantially focused in asingle direction. In one embodiment, the radiation then passes throughan IR cut filter 216, which blocks selective portions of the radiationso as to form a pattern. (In one embodiment, the filter 216 may also bereferred to as an IR-PASS filter, which operates as a physical shutter;the IR-PASS filter defines the image pattern of the beam) The patternformed by the IR cut filter 216 may have any configuration or shapewhich may be useful for generating a beam of radiation whose reflectedimage may be analyzed for perspective distortion. In variousembodiments, the pattern formed by the IR cut filter may be triangular,grid-like, circular, etc. Radiation passing through the IR cut filter isfurther focused by a lens 218, resulting in a patterned radiation beam219 that is emitted from the IR projector 208. In another embodiment, nofilter is used so that the radiation is emitted without a specificpattern.

The IR radiation beam generally expands in cross-sectional area asdistance from the projector 208 increases. In other words, a width ordiameter of the beam increases as distance from the projector 208increases. When the radiation beam 219 strikes a surface, it generatesan IR projection of its pattern on the surface. In one embodiment, theprojector is an infrared narrow beam projector which projects aninvisible near-IR spectrum beam pattern.

The IR projection generated when radiation beam 219 strikes a surface iscaptured as an IR image by an IR camera 220. The IR camera 220 issubstantially aligned with the IR projector 208 so as to be directedtowards an IR projection generated when the radiation beam 219 strikes asurface. The IR camera 220 includes lens 224 which focuses radiationonto an IR image sensor 222. In one embodiment the lens 224 has arelatively narrow field of view so as to primarily capture the IRprojection while generally excluding other portions of a scene. The IRimage sensor receives IR radiation and generates a digital output of IRimage data. In one embodiment, the IR camera includes a CMOS camerasensor with QVGA resolution at 60 frames per second (fps). In otherembodiments, the camera may employ other imaging technologies (e.g. CCDsensor) at different resolutions and having higher or lower frame rates(.e.g. 120 fps, 240 fps), etc.

The motion capture subassembly 206 also includes an RGB camera 226 forcapturing RGB images. The RGB camera 226 includes a wide-angle lens 230to provide a large field of view. In one embodiment, the wide-angle lensis a fish-eye lens. In other embodiments, multiple wide-angle lenses areutilized so as to provide greater field of view coverage. In otherembodiments, different types of lenses may be utilized in combination,such as a fish-eye lens and other lenses. The wide-angle lens 230focuses light onto an RGB image sensor 228, which generates a digitaloutput of RGB image data. In one embodiment, the RGB camera has QVGAresolution at 60 fps. In one embodiment, the pointing direction of theRGB camera can be adjusted by the user via mechanical means eithermanually or through actuators as are known in the art.

With reference to FIG. 4A, a controller 100 is shown with an IRprojector 208 emitting a patterned radiation beam 219 towards a surfacedenoted by X and Y axes. As the radiation beam 219 strikes the surface,an IR projection 232 of the IR beam 219 is generated on the surface. Theradiation beam 219 as shown has a circular cross-section, and expands asdistance from the IR projector 208 increases. Thus, the radiation beam219 is frustoconical in shape. The deviation of the central axis of theradiation beam 219 away from orthogonal to the surface is shown as anangle θ. When the central axis of the radiation beam 219 is orientedorthogonal to the surface (θ=0 degrees), then there is no distortion ofthe pattern of the beam 219 in the projection 232. In other words, theprojection will simply be a circle whose diameter varies as a functionof distance from the projector 208, increasing as distance increases.

However, when the radiation beam 219 is not oriented orthogonal to thesurface, then the projection 232 exhibits distortion of thecross-sectional pattern of the beam created by the angled orientation ofthe radiation beam 219. As the angle 0 increases from zero degrees, sothe distortion of the cross-sectional pattern of the beam exhibited bythe projection 232 becomes more exaggerated. Because the radiation beam219 in the present example is frustoconical, the projection 232 will bedistorted to be elliptically shaped when the axis of the radiation beam219 is not orthogonal to the surface. The eccentricity of the ellipticalshape will increase as θ increases from zero degrees, and the sizeincreases as the distance of the IR projector 208 from the projection232 increases.

The projection 232 is captured as an IR image by the IR camera 220. Byanalyzing the size, shape, orientation and intensity of the projectionin the captured IR image, it is possible to determine certain aspects ofthe orientation and position of the controller 100. The angle θ can bedetermined based on the shape of the projection 232 (θ increases as thedistortion increases), and the distance from the projection 232 to theIR projector 208 can be determined based on the size of the projection232. Once the angle θ and the distance from the projection 232 to theprojector 208 are known, the distance Z from the projector 208 to theplane of the surface can be determined as cos(θ) multiplied by thedistance from the projection 232 to the projector 208.

By analyzing the orientation of the pattern of the projection 232 it ispossible to determine the orientation of the controller in certainrelative terms. However, it is not possible to determine the orientationof the controller in terms of pitch, yaw, and roll, in an absolutesense. For example, the projection 232 generated by a controller 100having a 45 degree pitch and 90 degree roll would appear the same asthat of a controller 100 having a negative 45 degree yaw, as captured bythe IR camera 220. However, relative to the surface it is possible todetermine the deviation from orthogonal, and orientation, as well ascertain relative movements of the controller 100 over time.

In one embodiment, the captured IR image pattern may be used forcalibration of fish-eye distortion if a fish-eye lens is used as part ofthe motion capture subassembly.

FIG. 4B shows a cross-sectional view of a conically shaped patternedradiation beam 219 generated by a controller 100 striking a surface 234.Note that the relative dimensions of the beam 219 have been exaggeratedfor illustrative purposes only. The axis of the beam is orthogonal tothe surface 234, so as to produce a projection 232 whose image ascaptured by IR camera 220 is shown at 236. Because the beam is conicallyshaped and orthogonal to the surface 234, the projection is circular, asshown in the image 236. In order to facilitate evaluation of the image236 to determine orientation of the controller 100, the pattern of thebeam 219 is configured so as to be symmetric across no more than oneaxis.

FIG. 4C shows the beam 219 deviated by an angle θ from an orthogonal tothe surface 234. As a result, the projection 232 is distorted so as tobe elliptically shaped. In the image 236, the pattern of the projection232 appears distorted (elongated and shifted in the image field of view)accordingly.

FIG. 4D illustrates the beam 219 in a similar configuration as thatshown at FIG. 4C; however, the controller 100 has been rotated by 90degrees. Hence, the projection 232 is distorted in the same manner dueto the angle θ of deviation from orthogonal, but the orientation of thedistorted pattern in the projection is different as seen in the image236.

In sum, because the projection 232 will be distorted in predictable waysat various distances and angles of incidence, it is possible todetermine for the controller 100 its angle of deviation from orthogonal,distance from the surface, and relative orientation, based on analysisof the IR image 236 taken of the IR projection 232. While the foregoingexample has been described with reference to a conically shaped IR beamhaving a specific pattern, the same principles as those discussed abovemay be applied to a beam having any of various other shapes andpatterns, without departing from the scope of the present invention.

Furthermore, by tracking and analyzing the IR projection 232, therelative motion and changes in orientation of the controller 100 can bedetermined in accordance with the principles and methods describedabove.

FIG. 4E illustrates a controller 100 emitting an IR beam 219 having atriangular patterned cross-sectional shape. It is recognized that the IRbeam 219 may have any of various shapes and patterns without departingfrom the scope of the present invention. In some embodiments, thepattern of the IR beam will be wholly asymmetrical along any axis. Inother embodiments, the pattern of the IR beam is symmetrical along nomore than one axis.

In accordance with various embodiments of the invention, it isrecognized that where the focal point of the beam 219 and the focalpoint of camera 220 are the same or substantially the same, then thedifference in captured images of the projection 232 as the controller100 is moved to different positions and orientations may not bedetectable. Therefore, in embodiments of the invention, the projector208 and the camera 220 are arranged so that their focal points do notcoincide, but rather are offset from one another. The offset issufficient to permit detection of differences in captured images of theprojection 232 taken by the camera 220 as the controller is moved todifferent positions and orientations. In one embodiment the projector208 and camera 220 may be arranged so that their focal points differ interms of distance as measured along the axis of the beam 219. In anotherembodiment, the focal points are offset laterally from one anotherrelative to the axis of the beam 219. In still another embodiment, thefocal points differ both along the axis of the beam 219 and laterally,as well.

As noted with reference to FIG. 3A, the controller 100 includes an RGBcamera 226. In accordance with embodiments of the invention, the RGBcamera 226 is used to determine absolute position of the controller 100by tracking the display 106. More specifically, images of the display106 captured by the RGB camera 226 will demonstrate perspectivedistortion depending on the position of the controller 100. When thecontroller 100 (and by extension, the camera 226) moves closer to thedisplay 106, the display will appear larger in the captured image;whereas when the controller 100 moves farther from the display 106, thedisplay will appear smaller in the captured image. As the controller ismoved to different locations and oriented at various pitch, yaw and rollvalues, the display will appear in various parts of the field of viewand exhibit perspective distortion in a predictable manner. As such, thelocation and orientation of the controller 100 may be determined byanalyzing the captured image of the display 106 taken by the RGB camera226.

To facilitate determination of absolute position of the controller 100by the RGB camera 226, a calibration is performed in which thedimensions of the display 106 are ascertained. Once the dimensions areknown, it is possible to determine the position and orientation of thecontroller by analyzing captured images of the display 106 taken by RGBcamera 226. In one embodiment, the dimensions of the display 106 aremanually entered by a user of the interactive program. The dimensionsmay be entered numerically, selected from a list of options, orotherwise entered using any of various textual and graphical interfacesas are known in the art. In one embodiment, a user selects from amongstvarious aspect ratios and screen diagonal sizes. In another embodiment,the user selects from various display resolutions and screen diagonalsizes.

With reference to FIG. 5, an example of a user interface for enteringthe dimensions of a display 106 is shown. It should be understood thatmany different styles or configurations of user interfaces are possible,the presently described embodiment being merely one illustrativeexample. The user interface includes an aspect ratio selection menu 240from which a user may select the appropriate aspect ratio. Common aspectratios of displays are included such as 4:3, 3:2, and 16:9. If a userselects “other” then the user will be able to numerically specify adifferent aspect ratio. Also, a diagonal length measurement entry box242 is included wherein a user may enter the diagonal length of thedisplay 106. The unit length may be entered in inches, as shown, orusing other units of measurement, such as centimeters, meters, feet,etc. In still other embodiments, the dimensions of the screen can beobtained by the computing device (e.g., console) by way of a softwarerequest or look-up based on the type of screen that is connected to thecomputing device. This may occur automatically or at the direction of auser.

In another embodiment, the dimensions of the display are determined byutilizing the IR functionality of the controller 100 in conjunction withthe RGB camera 226. First, the display 106 is caused to display a blackimage, or some other still image or pattern for calibration purposes.Then, the user points the controller towards the display. The IRprojector 208 emits a near-IR/IR radiation beam which produces an IRprojection of the beam pattern on the display 106. The IR camera 220captures an IR image of the IR projection, which is then analyzed todetermine the distance Z of the controller 100 from the display 106 andthe relative orientation of the controller 100 based upon the distortionand size of the IR projection. The RGB camera simultaneously captures animage of the display, and the RGB image is analyzed to detect the outerframe of the display 106. As the distance Z and relative orientation ofthe controller 100 are known from the IR image analysis, the RGB imagecan be analyzed to determine the dimensions of the display 106. Morespecifically, the dimensions can be determined by analyzing the size andperspective distortion of the display in the RGB image in view of theknown distance Z and relative orientation of the controller 100.

In one embodiment, prior to detection of the outer frame of the displayin the RGB image, the display 106 is configured to render a pattern soas to aid in the detection of the outer frame of the display 106. Forexample, the display 106 may render a single color which contrasts withthe color of the outer frame of the display 106. Or the display 106 maybe caused to render bars along the borders of its screen, thushighlighting the borders and the corners of the display 106. In oneembodiment, the bars exhibit at least two different colors or patternswhich indicate the top, bottom, left and right sides of the display 106.In one embodiment, the display 106 is caused to render a pattern whichhighlights the corners of the display. In one embodiment, the display106 renders a pattern of a preset number of segments which can serve asaids in determining the outer frame of the display 106 as well as itsdimensions. In other embodiments of the invention, the display 106 maybe configured to render any type of pattern or image which is suitablefor aiding detection of the outer frame of the display 106 by analysisof an RGB image of the display.

With reference to FIG. 6A, one example of a pattern for aiding indetection of the outer frame of a display 106 is shown, in accordancewith an embodiment of the invention. The display 106 includes an outerframe 244 and a screen 245 which constitutes the display area of thedisplay 106. Bars 246 a, 246 b, 246 c, and 246 d are located at the top,bottom, left and right sides of the display 106, respectively, and thushighlight those sides of the outer frame 244. The bars 246 a-246 d mayhave the same color or pattern, or may have different colors orpatterns. In one embodiment, each of the bars has a slightly differentcolor to reduce the visual impact. Moreover, the colors may change inreal-time in accordance with colors being displayed on the display so asto reduce the visual impact. This color information may be sent to thecontroller to improve detection accuracy by color correlation. Cornerareas 248 a, 248 b, 248 c, and 248 d are regions specifically located atthe top left, top right, lower left, and lower right corners of thescreen 245. These corner areas 248 a-248 d thus highlight the corners ofthe outer frame 244. The corner areas 248 a-248 d may have the samepattern or color, or may have different patterns or colors. By providinga preset display pattern on the screen 245, such as the foregoing barsand corner areas, an RGB image taken of the display 106 can be analyzedto determine the location of the preset display pattern. And withreference to the preset display pattern in the RGB image, which definesthe boundary between the screen 245 and the outer frame 244, the outerframe 244 of the display 106 can then be detected more easily.

With reference to FIG. 6B, an example of a pattern for aiding indetection of the outer frame of a display 106 is shown, in accordancewith an embodiment of the invention. As shown, the pattern displayed onthe screen 245 of the display 106 is a grid-like pattern 250. Such apattern may serve to highlight and better enable detection of the outerframe 244 of the display 206. Additionally, the use of such a patternmay also help to reveal distortions inherent in the screen 245.

To determine the position, orientation, and movement of the controller100 during activity with the interactive program, the controller 100utilizes its RGB camera 226 to track the outer frame of the display. Asthe controller is moved to different positions and orientations, so thesize, shape, and orientation of the display 106 in images captured bythe RGB camera 226 changes accordingly. These changes in the RGB imageof the display 106 are the result of both perspective distortion and theoptical qualities of the lens 230 in the RGB camera 226.

With reference to FIGS. 7A, 7B, and 7C, overhead views of the controller100 at various locations in relation to the display 106 are shown. InFIG. 7A, the controller 100 is located in front of the display 106, suchthat the display 106 appears in a captured image 252 taken by the RGBcamera 226 as shown. In FIG. 7B, the controller 100 has been movedcloser to the TV. The result as shown in the captured image 254 is thatthe display 106 appears larger in the image when the controller 100 iscloser to the display 106. At FIG. 7C, the controller 100 has been movedto the left side of the display 106. As a result, the image of thedisplay 106 as seen in the captured image 256 exhibits perspectivedistortion resulting from the location of the controller 100. The leftside of the display 106 appears taller in the image 256 than the rightside of the display 106, and the overall appearance of the display 106appears to be shortened in width. The perspective distortion effectsresulting from movement of the controller 100 are predictable, and assuch, the location of the controller 100 can be determined by examiningcaptured images of the RGB camera 226 and analyzing the perspectivedistortion of the display.

With reference to FIG. 7D, an overhead view of the controller 100 inrelation to the display 106 is illustrated, wherein the yaw of thecontroller 100 has been changed in comparison to that of FIG. 7A.Specifically, the yaw of the controller 100 has been negatively shifted(shifted to towards the left) relative to the display 106. The result isthat in the captured image 258, the display appears shifted towards theright side of the image. Moreover, the display exhibits some perspectivedistortion wherein the left side of the display appears shorter than theright side.

With reference to FIG. 7E, an overhead view of the controller 100 inrelation to the display 106 is illustrated, wherein the roll of thecontroller 100 has been changed in comparison to that of FIG. 7A.Specifically, the roll of the controller 100 has been positivelyadjusted (clockwise rotation) relative to the display 106. The result isthat in the captured image 260, the display appears to be tilted in acounterclockwise fashion.

With reference to FIG. 7F, a side view of the controller 100 in relationto the display 106 is illustrated, wherein the pitch of the controller100 has been changed in comparison to that of FIG. 7A. Specifically, thepitch of the controller 100 has been positively shifted (shiftedupwards) relative to the display 106. The result is that in the capturedimage 262, the display appears shifted towards the bottom of the image.Moreover, the display exhibits some perspective distortion wherein thetop side of the display appears shorter than the bottom side.

As can be seen from the foregoing embodiments, the position andorientation of the controller 100 relative to the display 106 can bedetermined by capturing and analyzing RGB images of the display capturedby the RGB camera 226 at the controller 100. The appearance of thedisplay 106 within the captured RGB images will change in accordancewith changes in the position, yaw, pitch, and roll of the controller 100relative to the display 106.

In various embodiments of the invention, the appearance of the display106 as captured by the RGB camera 226 is tracked by specificallytracking the outer frame of the display 106. The outer frame of manydisplays is generally of a uniform color (often a dark color such asblack or gray) which is conducive to tracking. Additionally, manydisplays may have a small outer boundary of the display area which isnot illuminated, which may also be tracked. Furthermore, the display mayinclude characteristic design items such as a logo or light indicatingthat the display is operating which remain stationary and uniform duringoperation. These design items may provide additional features which canbe tracked for the purpose of determining the position and movement of acontroller.

With reference to FIG. 8A, a display 106 is shown, wherein a series oftracking bars 264 are shown at the borders of the screen 245 of thedisplay 106. In accordance with an embodiment of the invention, thetracking bars 264 are tracked in RGB images of the display taken by theRGB camera 226 on the controller 100. By tracking the bars 264 from thecontroller 100, and analyzing images of the display to determineperspective distortion, location, and orientation of the tracking bars,so the position, orientation, and movement of the controller 100 isdetermined. In some instances, the outer frame of a display may not bereadily ascertainable, such as when a projector is utilized to projectan image onto a wall or a projection screen. Therefore, it may bedesirable to generate a border which can be tracked such as the thinbars 264.

The tracking bars 264 may be in the form of thin lines having athickness determined based on a trade-off between accuracy of detectionand visual impact. Thicker lines may promote better detection andtracking of the display, but negatively impact the visual display.Whereas thinner lines will improve visual appearance but may be moredifficult to track because of their diminished appearance.

In various embodiments, the colors or shades of the tracking bars 264may be different from one another so as to reduce the visual impact ofthe bars. Moreover, the colors of the tracking bars may be altered inreal-time depending on the colors of the video content being displayedon the display 106, so as to reduce the visual impact of the trackingbars 264 relative to the video content. In order not to negativelyimpact the detection of the tracking bars at the controller, thecontroller may be updated real-time with information about the currentcolor of the tracking bars. In one embodiment, the controller includes acolor table which contains colors correlated to the colors that thetracking bars 264 may exhibit. In captured images of the display at thecontroller, the captured images may be evaluated for the presence of thecolors and shape of the tracking bars, thereby facilitatingdetermination of the controller's position and movement.

In one embodiment, in order to create a more natural look, the trackingbars form a rectangular color frame, within which the video image isdisplayed. The rectangular color frame may be grey scale or black, so asto minimize its visual impact.

In other embodiments of the invention, the outer frame of a display maybe altered so as to aid in its detection for purposes of tracking acontroller. For example, with reference to FIG. 8B, a display 106 isshown having reflective stickers 265 a, 265 b, 265 c, and 265 d attachedto the corners of the display 106. The reflective stickers 265 a-265 dprovide areas of the outer frame which may be tracked in RGB images ofthe display captured by the controller's RGB camera. To further enhancethe ability to track the reflective stickers 265 a-265 d, they may havespecific patterns or colors, and be designed to contrast with the colorof the display's outer frame 244. The reflective stickers may havedifferent patterns or colors from each other so as to indicate thecorner of the display to which they are attached.

The controller 100 is intended to be maneuvered so as to enable a userto interface with an interactive program. As the positional tracking ofthe controller is dependent upon the controller 100 being able to detectthe display 106, it is desirable for the RGB camera of the controller100 to have as wide a field of view as possible. In various embodiments,this may be accomplished by the inclusion of multiple RGB cameras, and afish-eye lens.

With reference to FIG. 9A, a controller 100 is shown in accordance withan embodiment of the invention. The controller 100 as shown includes RGBcameras 266 a, 266 b, and 266 c. The field of view of RGB camera 266 aoverlaps with the field of view of RGB camera 266 b, and the field ofview of RGB camera 266 b overlaps with that of RGB camera 266 c. Bycombining the fields of view from the multiple RGB cameras 266 a-266 c,the total field of view available for use by the controller 100 for RGBimage capture is in the range of 270 degrees. While the RGB cameras 266a-266 c as shown are aligned in substantially the same plane, additionalcameras may be placed so as to be aligned in planes intersecting ortransverse to that shown in FIG. 9A.

With reference to FIG. 9B, a controller 100 having a 360 degree field ofview is shown, in accordance with an embodiment of the invention. Thecontroller 100 as shown includes multiple RGB cameras 268 a, 268 b, 268c, 268 d, 268 e, and 268 f. The fields of view of each of these camerasoverlaps with the fields of view of two of the other RGB cameras, so asto collectively form a 360 degree field of view.

With reference to FIG. 10A, a controller 100 having an RGB camera 270which includes a fisheye lens is shown. A fisheye lens providesextremely wide viewing angles, potentially in the range of approximately270 degrees. Thus, the use of a fisheye lens enables a very wide fieldof view to be captured by the RGB camera 270. As is known in the art,fisheye lenses produce a fair amount of distortion in order to capturesuch a wide field of view onto a planar space such as an image sensor.Therefore, in embodiments of the invention which incorporate a fisheyelens into an RGB camera, the analysis of the RGB images captured by theRGB camera takes into account the distortion inherent in images producedusing a fisheye lens.

With reference to FIG. 10B, a controller 100 is shown having two RGBcameras 272 a and 272 b, each of which incorporates fisheye lenses. Thecameras 272 a and 272 b are pointed in opposite directions. The fieldsof view of the cameras 272 a and 272 b overlap so as to provide a 360degree field of view for the controller 100.

In various embodiments of the invention, the IR functionality of thecontroller 100 may be combined with the RGB image capture duringreal-time tracking of the controller 100. For example, in oneembodiment, the continuous tracking of the display via the RGB camera isutilized as the primary method of determining the position and movementof the controller. However, when the position of the controller cannotbe obtained by this method, such as when the controller is moved so thatthe display is no longer in the field of view of the RGB camera, thenthe IR projector and IR camera may be initiated to enable determinationof relative motion and orientation.

As shown at FIG. 11, a controller 100 located at position A is orientedfacing the display 106 such that the display 106 appears in the field ofview of its RGB camera. The position, movement and orientation of thecontroller 100 may thus be determined by tracking the display 106 incaptured RGB images of the display. However, when the controller 100 ismoved to position B, the controller is now so close to the display 106that the outer borders of the display are no longer in the field of viewof the RGB camera. As such, the position of the controller 100 cannot bedetermined based on analysis of the captured images of the RGB camera.In order to continue tracking of the position and orientation of thecontroller 100, the IR projector and IR camera of the controller 100 areturned on. In accordance with the principles discussed above, the IRprojector generates a projection which is captured by the IR camera as aseries of IR images. Since the controller 100 at position B is so closeto the display 106, the projection is likely on the surface of thedisplay 106 itself, though it may be on another surface such as a wall.Analysis of the captured IR images of the IR camera yields thecontroller's distance from the projection surface and the orientation ofthe controller despite the loss of the ability to determine positionbased on RGB images. In particular, the orientation can be determined byutilizing the last known orientation of the controller according to RGBimage analysis and thereafter initiating the IR functionality andcontinuously tracking the changes in the IR projection.

A similar situation occurs when the controller 100 is moved to aposition C, wherein the controller 100 is oriented in the direction ofsidewall 274, such that the display 106 is outside the field of view ofthe controller's RGB camera. Because the display 106 is outside thefield of view of the controller's RGB camera, the IR projector and IRcamera are initiated to enable determination of position and orientationof the controller while it is directed away from the display 106.

Additionally, the IR functionality of the controller 100 may be usefulas a supplement to the RGB image-based position/orientationdetermination when better tracking of the controller is desired. Forexample, with continued reference to FIG. 11, when the controller 100 ismoved away from the display 106, such as at position D, the display 106may appear rather small in the field of view of the controller's RGBcamera. As such, tracking the display may become more difficult, and thesystem may be prone to less robust tracking of the display. Therefore,the IR-based tracking mechanism is initiated to supplement the RGB imagetracking mechanism for improved tracking of the controller, and as aredundant system in case the position/orientation of the controllerbecomes non-ascertainable based on RGB image analysis alone.

When the controller is moved so as to cause the display 106 to move outof the field of view of the controller's RGB camera, the shift of thedisplay in the field of view of the RGB camera occurs over time. Atfirst, one or two corners of the display will be lost from the field ofview, followed by the remaining corners of the display. Because theorientation of the display in the field of view is known before thecorners are lost from view, and because the dimensions of the displayare known, it is possible to continue to determine the position of thecontroller even while only two corners of the display remain in thefield of view of the RGB camera. However, since the loss of controllerposition determination based on RGB images is imminent at this point, itmay be desirable to turn on the IR projector and IR camera before morethan two corners of the display are lost from the RGB camera's field ofview.

With reference to FIGS. 12A, 12B, and 12C, an overhead view of acontroller 100 is shown as it is maneuvered away from a display 106. AtFIG. 12A, the controller is facing the display 106, such that thedisplay 106 is fully within the field of view of the controller's RGBcamera. This may be seen in a captured RGB image 276 of the RGB camera,wherein the entirety of the display is visible in the RGB image 276. AtFIG. 12B, the controller 100 has been turned towards the side, such thatonly the two right corners of the display are now visible in the RGBcaptured image 278. Because the dimensions of the display 106 are known,and its orientation was known before movement of the display out of theRGB camera's field of view, it is still possible to determine theposition and orientation of the controller even when only two corners ofthe display are in the field of view. However, loss of the entirety ofthe display from the field of view is imminent, and so the IRfunctionality of the controller 100 is initiated in anticipation of thisoccurring. In this manner, the transition from the RGB image basedmechanism to the IR mechanism is smoothed. At FIG. 12C, the controller100 is oriented such that the display 106 is completely outside thefield of view of the controller's RGB camera, and does not appear in acaptured image 280 of the RGB camera. At this point, the system reliesupon the IR projector and IR camera of the controller to determineposition and orientation of the controller 100.

With reference to FIG. 13, a user 102 holding a controller 100 is shown,in accordance with an embodiment of the invention. When the user 102holds the controller at position D, the controller 100 is approximatelydirected towards the display 106 such that the display appears in thefield of view of the controller's RGB camera. However, when the usermoves the controller to position E, the controller 100 is directed awayfrom the display 106. Thus, an IR projector and IR camera of thecontroller 100 are initiated and utilized to determine the position andorientation of the controller 100.

With reference to FIG. 14, a logical diagram illustrating a system forproviding controller input to an interactive program is shown. Thecomputer 103 executes an interactive program 298 which can receiveinteractive input from the controller 100. The interactive program isdisplayed to a user via display logic 306, which sends video data to thedisplay 106 for rendering. The controller 100 includes a wirelesstransceiver 320 for facilitating wireless communications with a computer103, which also includes a wireless transceiver 304. The controller 100includes an IR projector and IR camera controller 308, which controlsthe on/off state of the IR projector and IR camera of the controller inaccordance with commands received from the computer 103. For example,the IR projector and IR camera may be turned on for initializationpurposes in order to determine the dimensions of the display 106. Or theIR projector and IR camera may be turned on when the display 106 isoutside or nearly outside the field of view of the RGB camera of thecontroller 100. The IR projector and IR camera may also be triggered onwhen better tracking of the controller 100 is desired, even when thedisplay 106 is within the field of view of the RGB camera, such as whenthe controller 100 is located at a large distance from the display 106.

IR image capture logic 310 controls the IR image capture by the IRcamera, providing a continuous stream of IR images at a regular framerate such as 60 frames per second. In various embodiments, the framerate of the IR image stream may be higher than 60 frames per second(e.g. 120 fps) or lower than 60 frames per second. An IR image analyzer312 is included for analyzing the stream of captured IR images. Asdescribed above, the IR camera is directed so as to capture images ofthe IR projection generated by the IR projector. Thus, the IR imageanalyzer 312 determines the shape, size, and orientation of the IRprojection in the captured IR images. Based on this analysis, thedistance from the projection surface, and the relative pitch and yaw (orrelative roll) of the controller 100 is determined

The controller 100 additionally includes an RGB camera controller whichcontrols the on/off state of the controller's RGB camera. RGB imagecapture logic 316 is provided for controlling the RGB image capture ofthe controller's RGB camera, providing a continuous stream of RGB imagesat a regular frame rate such as 60 frames per second. In variousembodiments, the frame rate of the RGB camera may be greater than orless than 60 frames per second. A higher frame rate (e.g. 120 fps)yields greater fidelity for purposes of position and orientationdetermination. An RGB image analyzer 318 performs an initial processingof the captured RGB image frames. In one embodiment, the RGB imageanalyzer determines the location and shape of the display within thecaptured RGB images. In one embodiment, this data is reduced tocoordinates within the space of the RGB images which describe theoutline of the display. These coordinates are sent to the computer 103for further processing, as the determination of the position/orientationof the controller references the dimensions of the display 106. Inanother embodiment, the RGB image analyzer performs a hash orcompression of the captured RGB images. The compressed RGB images aresent to the computer 103 for analysis to determine the location andshape of the display in the images.

The computer 103 includes an RGB data analyzer 296, which analyzes dataregarding the captured RGB images from the controller 100. Morespecifically, the location, shape, size, and orientation of the displaywithin the captured RGB images is determined with reference to theactual dimensions of the display 106. As such, the position andorientation of the controller 100 is determined by the RGB data analyzer296 based on the perspective distortion and orientation of the displaywithin the captured images, and by reference to the actual dimensions ofthe display. The position is determined in 3-D space relative to thedisplay 106, and the orientation of the controller 100 is determined interms of pitch, yaw, and roll.

While the illustrated embodiment contemplates processing of the RGB dataanalyzer 296 at the computer 103, in another embodiment, the functionsof the RGB data analyzer 296 are carried out at the controller 100. Thecontroller 100 may include DSP hardware and memory which are required toenable processing of image data as well as final determination of thecontroller's position and orientation. The controller 100 thus may beconfigured to output to the computer 103 its location relative to thedisplay 106.

The computer 103 includes calibration logic 290 for calibrating thecontroller by determining the dimensions of the display 106. Acalibration display unit 292 causes the computer 103 to output to thedisplay 106 a blank image, or an image with a specific color or patternfor calibration purposes. For example, in accordance with theabove-described methods of calibration, the display may be black ordarkened, or may include colored bars and the like which highlight theouter frame and corners of the display 106. A calibration data analyzer294 determines the dimensions of the display 106 based on data receivedfrom the controller 100. The data received from the controller 100 mayinclude IR-based distance/orientation information of the controller 100,and an RGB image including the display as captured by the controller'sRGB camera. Based on such data received from the controller 100, thecalibration data analyzer determines the dimensions of the display 106.

An interactive program 298 runs on the computer 103. In accordance withan embodiment of the invention, the interactive program 298 includesdynamic zone determination 300, which, based on the operation of theinteractive program, determines three dimensional zones which affect thefunctionality of the controller 100. Position-based function trigger 302determines the function to be executed by the interactive program basedon received input from the controller, and the controller's positionrelative to the zones determined by the dynamic zone determination 300.

The above-described system of FIG. 14 constitutes means for determiningposition/orientation/movement of the controller 100. Means forinitializing the computer-based system to establish a starting positionof the controller device relative to the display are provided by thecalibration logic 290. Means for capturing an image of the display usinga camera integrated in the controller device are provided by the RGBcamera controller 314. Means for capturing an image of a radiationpattern projected by the controller device are provided by the IR cameracontroller 308. Means for analyzing the captured image of the displayand the image of the radiation pattern projected by the controllerdevice to determine a size of the display in the captured image at adistance of the controller device as determined by the image of theprojected radiation pattern are provided by the calibration dataanalyzer 294. Means for capturing successive images of the display atthe controller device are provided by the RGB camera controller 314.Means for determining a position of the controller device relative tothe display based on a perspective distortion of the display in thecaptured successive images of the display are provided by the RGB imageanalyzer 318 and the RGB data analyzer 296. Means for providing thedetermined position of the controller to the computer-based system tointerface with the interactive program to cause an action by theinteractive program are provided by the RGB data analyzer 296 as itprovides data to interactive program 298. Means for determining anorientation of the controller device based on a location and orientationof the display in the captured image of the display are provided by theRGB data analyzer 296. Means for determining movement of the controllerdevice by tracking the position of the controller at the time of eachcaptured successive image are provided by the RGB image analyzer 318 andthe RGB data analyzer 296. Means for determining position of claim 17,wherein the radiation pattern projected by the controller is generatedby an IR projector.

Means for auxiliary position determination are provided by the IRprojector and camera controller 308, the IR image capture logic 310, andthe IR image analyzer 312. The means for auxiliary positiondetermination are configured to perform the following operations whenthe position of the controller device cannot be determined based on thecaptured successive images of the display: (1) project the radiationpattern from the controller device; (2) capture successive images of theradiation pattern at the controller device; and (3) estimate theposition of the controller device based on the captured successiveimages of the radiation pattern.

In various embodiments, it is not necessary to capture images of theentirety of the display, so long a portion of the display sufficient toenable functionality of the controller is captured. Therefore, in oneembodiment, the means for determining position is configured so that thecapturing an image of the display includes capturing an image of aportion of the display sufficient to enable determination of the size ofthe display. In one embodiment, the means for determining position isconfigured so that the capturing successive images of the displayincludes capturing successive images of a portion of the displaysufficient to enable determination of the position of the controllerdevice.

In one embodiment, the controller 100 may consist of a handle and aseparate attachment which provides expanded capabilities. FIG. 15illustrates the components of a handle 424 of a controller withexpansion connector 402, in accordance with an embodiment of theinvention. Although controllers defined within the spirit and scope ofthe claims may have more or less components, these exemplary componentsshow example electronics, hardware, firmware, and housing structure todefine an operable example. These example components, however, shouldnot limit the claimed inventions, as more or fewer components arepossible. Handle 424 is configured to be held by a user operatingcontroller 100 with a single hand. A user's second hand may, of course,be used to hold or select buttons on handle 424. A user holdingcontroller 100 can provide input by pressing buttons, such as top button410 and bottom button 408. In one embodiment input can also be providedby moving the controller within a three-dimensional space when anattachment is coupled to handle 424, such as the one shown in FIG. 17A.Controller 100 is configured to operate wirelessly, which facilitatesfreedom of controller movement in order to interact with the computer103. Wireless communication can be achieved in multiple ways, such asvia Bluetooth® wireless link, WiFi, infrared (not shown) link, etc.

Attachments providing expanded capabilities to handle 424 are connectedand disconnected to expansion connector 402. In one embodiment, anattachment enables the base computing device to locate the combinationof handle and attachment within a three-dimensional space via visualrecognition of images taken by a camera within the attachment itself.More specifically, and as explained previously, the location of thecombined handle and attachment is determined from images taken at thecontroller 100 based on perspective distortion and orientation of thedisplay 106 in the captured images. Other embodiments provide additionalcommunication capabilities to controller 100, such as an attachment thatprovides ultrasonic communication with the computer 103 or with othercontrollers in the field of play. In yet another embodiment, anattachment provides infrared capabilities to allow the controller tocommunicate via infrared frequencies with the computer, or to usecontroller 100 as a remote control for a TV or other electronicequipment.

In one embodiment, the attachment communicates directly with thecomputer and can act upon commands received from the computer, such asturning on an internal light or emitting a sound. In another embodiment,the attachment is directly controlled by handle 424 and the attachmentonly reacts to commands from handle 424. In yet another embodiment, theattachment can react to commands received from the computer or from thehandle.

Inside handle 424, printed circuit board 416 holds processor 412,Input/Output (I/O) module 406, memory 416, and Bluetooth module 418, allinterconnected by bus 422. A Universal Serial Bus (USB) module 420 alsoprovides interactivity with the base computing device, or with otherdevices connected to USB port 432. The USB port can also be used tocharge the rechargeable battery 430. Vibrotactile feedback is providedby vibrotactile module 428. Speaker 426 provides audio output.

Note that the above controller configuration is exemplary and manymodifications thereto, including eliminating or adding modules, wouldoccur to a person of ordinary skill in the art with access to thepresent Specification, and is well within the scope of the claimedinvention. For example, controller 300 can also include sensors formechanical tracking of the controller movement.

FIG. 16 depicts a controller 100 with sensors for improving movementtracking, according to one embodiment. Different embodiments includedifferent combinations of sensors, such as magnetometers 434,accelerometers 436, gyroscopes 438, etc. An accelerometer is a devicefor measuring acceleration and gravity induced reaction forces. Singleand multiple axis models are available to detect magnitude and directionof the acceleration in different directions. The accelerometer is usedto sense inclination, vibration, and shock. In one embodiment, threeaccelerometers 436 are used to provide the direction of gravity, whichgives an absolute reference for 2 angles (world-space pitch andworld-space roll). Controllers can suffer accelerations exceeding 5 g,therefore accelerometers able to operate with forces exceeding 5 g areused inside controller 100.

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the controller. In one embodiment, threemagnetometers 434 are used within the controller, ensuring an absolutereference for the world-space yaw angle. The magnetometer is designed tospan the earth magnetic field, which is ±80 microtesla. Magnetometersare affected by metal, and provide a yaw measurement that is monotonicwith actual yaw. The magnetic field may be warped due to metal in theenvironment, which causes a warp in the yaw measurement. If necessary,this warp can be calibrated using information from the gyros (see below)or the camera. In one embodiment, accelerometer 436 is used togetherwith magnetometer 434 to obtain the inclination and azimuth of thecontroller.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes provide information about movement across the respective axis(x, y and z) based on inertial sensing. The gyroscopes help in detectingfast rotations. However, the gyroscopes can drift overtime without theexistence of an absolute reference. This requires, resetting thegyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of the display 106, accelerometer, magnetometer, etc. Ahand-held device can rotate faster than 500 degrees/sec, so a gyroscopewith a spec of more than 1000 degrees/sec is recommended, but smallervalues are also possible.

The information from the different sources can be combined for improvedlocation and orientation detection. For example, if the controller ismoved or oriented such that the display is no longer in the field ofview of the RGB camera, then the accelerometer's orientation sensing isused to detect that the controller is facing away from the display. Inone embodiment, controller 100 includes speaker 426 to provide audiofeedback to the player. The controller can produce a beep when thedisplay is not in the RGB camera's field of view, prompting the playerto orientate the controller in the right direction or to come back intothe field of play.

FIG. 17A depicts an attachment 502 for the handle 424 with a “rich”feature set. It should be appreciated that the embodiment illustrated inFIG. 17A is exemplary and other embodiments may include a subset of thefeatures of attachment 502. The embodiment illustrated in FIG. 17Ashould therefore not be interpreted to be exclusive or limiting, butrather exemplary or illustrative.

The different modules in spherical attachment 502 are interconnected viaa common bus, but other interconnection mechanisms are possible.Connector 504 provides the interface to connect or disconnect attachment502 from the controller. Attachment 502 includes a processor or circuitplus memory allowing the attachment to process computer instructions.Further, attachment 502 includes communication modules such asultrasound, infrared, and WiFi. Such communications enable theattachment to communicate with the computer or other electronic devices,which is referred to herein as a communications interface between thecontroller and the computer or any other electronic device. In oneembodiment, the attachment operates as a modem by receiving informationfrom the controller and forwarding the information to the computer, andvice versa.

Information received by the attachment and passed to the controller isused to change the state of the controller. For example, the controllermay emit a sound, change button configuration, disable the controller,load registers in memory, send a command to the attachment to light up,etc. The information received by the computer is used by the interactiveprogram to update the state of the interactive program. For example, theinteractive program may move an avatar on the screen or change thestatus of the avatar, fire a weapon, start a game, select an option in amenu, etc.

An accelerometer, a magnetometer and a gyroscope provide mechanicalinformation related to the movement of the attachment. In oneembodiment, the mechanical or inertial information is combined withother location determination information, such as visual tracking of thedisplay, in order to refine the determination of the location of thecontroller-attachment combo.

An internal light emitting device allows the attachment to be lit fromthe inside to provide user feedback. In one embodiment, light emittingdevice can emit light of a single color, and in another embodiment,light emitting device can be configured to emit light from a choice ofcolors. In yet another embodiment, attachment 502 includes several lightemitting devices, each device being capable of emitting light of onecolor. The light emitting device is configurable to emit differentlevels of brightness. The computer can provide interactivity to the userholding the controller by changing the light emitting status ofattachment 502, producing audio signals, or with vibrotactile feedback,etc. One feedback operation or a combination of feedback operations ispossible. In one embodiment, the type of feedback is selected from alist of predefined interactivity, and based on what is occurring in agame.

A microphone and a speaker provide audio capabilities, while a batterypowers the rest of the components, including the processor and the lightemitting device. The battery can also be used by the handle as a secondsource of power. For example, if the rechargeable battery in thecontroller is discharged, the attachment can provide the required powerso the user can continue playing instead of having to stop to rechargethe controller. In one embodiment, attachment 502 does not include thebattery and power to the modules in attachment 502 is obtained via anelectrical connection with the power source of the handle.

An IR projector and an IR camera provide IR functionality fordetermining relative position and orientation of the controller asdescribed above. An RGB camera captures RGB images of the display sothat the position of the controller may be determined based on theperspective distortion and orientation of the display in the capturedRGB images.

A USB module allows USB communication to and from the attachment. In oneembodiment, the USB connection is used to charge the battery in theattachment. In yet another embodiment, attachment 502 includes files inmemory that are transferred to the controller, or to the computer, or toboth the controller and the computer. The files in memory can includeconfiguration files or programs that are transferred for execution inthe controller or the gaming system. The files can be used to identify aspecific user, to configure the controller or the base system, to load agame, to add features to existing games, etc. For example, one file is agame that is loaded to the computer for playing, another file containskaraoke songs that can be used in a sing-along game, another filecontains new player rosters and statistics for an update to a sportsgame, etc. In addition, the attachment can be used to store userparameters, such as player configuration for a particular game. Theplayer can then use the attachment in a different gaming system to playwith other players using the configuration obtained from the originalgaming system.

FIG. 17B illustrates an embodiment where the attachment of FIG. 17A isconnected to the controller of FIG. 15. In one embodiment, attachment502 interacts with controller 424 via a communications interface, suchas a USB interface. In another embodiment, attachment 502 is inelectrical communication with one or several internal modules insidecontroller 424. For example, processor/circuit of attachment 502 (asseen in FIG. 17A) is connected to bus 422 of controller 424 (as seen inFIG. 15), thus allowing the processor of attachment 502 to communicatewith the modules in the controller attached to the bus. The processor ofattachment 502 can access memory 416 to write or read data directly, orgenerate interrupts for processor/circuit 412 of controller 424 tosignal an external event which must be processed by processor 412.

It should be noted that the embodiment depicted in FIG. 17B is exemplaryand other embodiments may include fewer components.

FIG. 18 illustrates hardware and user interfaces that may be used todetermine controller location, in accordance with one embodiment of thepresent invention. FIG. 18 schematically illustrates the overall systemarchitecture of the Sony® Playstation 3® entertainment device, a consolethat may be compatible for interfacing a control device with a computerprogram executing at a base computing device in accordance withembodiments of the present invention. A system unit 500 is provided,with various peripheral devices connectable to the system unit 500. Thesystem unit 500 comprises: a Cell processor 528; a Rambus® dynamicrandom access memory (XDRAM) unit 526; a Reality Synthesizer graphicsunit 530 with a dedicated video random access memory (VRAM) unit 532;and an I/O bridge 534. The system unit 500 also comprises a Blu Ray®Disk BD-ROM® optical disk reader 540 for reading from a disk 540 a and aremovable slot-in hard disk drive (HDD) 536, accessible through the I/Obridge 534. Optionally the system unit 500 also comprises a memory cardreader 538 for reading compact flash memory cards, Memory Stick® memorycards and the like, which is similarly accessible through the I/O bridge534.

The I/O bridge 534 also connects to six Universal Serial Bus (USB) 2.0ports 524; a gigabit Ethernet port 522; an IEEE 802.11b/g wirelessnetwork (Wi-Fi) port 520; and a Bluetooth® wireless link port 518capable of supporting of up to seven Bluetooth connections.

In operation, the I/O bridge 534 handles all wireless, USB and Ethernetdata, including data from one or more game controllers 502-503. Forexample when a user is playing a game, the I/O bridge 534 receives datafrom the game controller 502-503 via a Bluetooth link and directs it tothe Cell processor 528, which updates the current state of the gameaccordingly.

The wireless, USB and Ethernet ports also provide connectivity for otherperipheral devices in addition to game controllers 502-503, such as: aremote control 504; a keyboard 506; a mouse 508; a portableentertainment device 510 such as a Sony Playstation Portable®entertainment device; a video camera such as an EyeToy® video camera512; a microphone headset 514; and a microphone 515. Such peripheraldevices may therefore in principle be connected to the system unit 500wirelessly; for example the portable entertainment device 510 maycommunicate via a Wi-Fi ad-hoc connection, whilst the microphone headset514 may communicate via a Bluetooth link.

The provision of these interfaces means that the Playstation 3 device isalso potentially compatible with other peripheral devices such asdigital video recorders (DVRs), set-top boxes, digital cameras, portablemedia players, Voice over IP telephones, mobile telephones, printers andscanners.

In addition, a legacy memory card reader 516 may be connected to thesystem unit via a USB port 524, enabling the reading of memory cards 548of the kind used by the Playstation® or Playstation 2® devices.

The game controllers 502-503 are operable to communicate wirelessly withthe system unit 500 via the Bluetooth link, or to be connected to a USBport, thereby also providing power by which to charge the battery of thegame controllers 502-503. Game controllers 502-503 can also includememory, a processor, a memory card reader, permanent memory such asflash memory, light emitters such as an illuminated spherical section,LEDs, or infrared lights, microphone and speaker for ultrasoundcommunications, an acoustic chamber, a digital camera, an internalclock, a recognizable shape such as the spherical section facing thegame console, and wireless communications using protocols such asBluetooth®, WiFi™, etc.

Game controller 502 is a controller designed to be used with two hands,and game controller 503 is a single-hand controller with an attachment.In addition to one or more analog joysticks and conventional controlbuttons, the game controller is susceptible to three-dimensionallocation determination. Consequently gestures and movements by the userof the game controller may be translated as inputs to a game in additionto or instead of conventional button or joystick commands. Optionally,other wirelessly enabled peripheral devices such as the Playstation™Portable device may be used as a controller. In the case of thePlaystation™ Portable device, additional game or control information(for example, control instructions or number of lives) may be providedon the screen of the device. Other alternative or supplementary controldevices may also be used, such as a dance mat (not shown), a light gun(not shown), a steering wheel and pedals (not shown) or bespokecontrollers, such as a single or several large buttons for arapid-response quiz game (also not shown).

The remote control 504 is also operable to communicate wirelessly withthe system unit 500 via a Bluetooth link. The remote control 504comprises controls suitable for the operation of the Blu Ray™ DiskBD-ROM reader 540 and for the navigation of disk content.

The Blu Ray™ Disk BD-ROM reader 540 is operable to read CD-ROMscompatible with the Playstation and PlayStation 2 devices, in additionto conventional pre-recorded and recordable CDs, and so-called SuperAudio CDs. The reader 540 is also operable to read DVD-ROMs compatiblewith the Playstation 2 and PlayStation 3 devices, in addition toconventional pre-recorded and recordable DVDs. The reader 540 is furtheroperable to read BD-ROMs compatible with the Playstation 3 device, aswell as conventional pre-recorded and recordable Blu-Ray Disks.

The system unit 500 is operable to supply audio and video, eithergenerated or decoded by the Playstation 3 device via the RealitySynthesizer graphics unit 530, through audio and video connectors to adisplay and sound output device 542 such as a monitor or television sethaving a display 544 and one or more loudspeakers 546. The audioconnectors 550 may include conventional analogue and digital outputswhilst the video connectors 552 may variously include component video,S-video, composite video and one or more High Definition MultimediaInterface (HDMI) outputs. Consequently, video output may be in formatssuch as PAL or NTSC, or in 720p, 1080i or 1080p high definition.

Audio processing (generation, decoding and so on) is performed by theCell processor 528. The Playstation 3 device's operating system supportsDolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and thedecoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera 512 comprises a singlecharge coupled device (CCD), an LED indicator, and hardware-basedreal-time data compression and encoding apparatus so that compressedvideo data may be transmitted in an appropriate format such as anintra-image based MPEG (motion picture expert group) standard fordecoding by the system unit 500. The camera LED indicator is arranged toilluminate in response to appropriate control data from the system unit500, for example to signify adverse lighting conditions. Embodiments ofthe video camera 512 may variously connect to the system unit 500 via aUSB, Bluetooth or Wi-Fi communication port. Embodiments of the videocamera may include one or more associated microphones and also becapable of transmitting audio data. In embodiments of the video camera,the CCD may have a resolution suitable for high-definition videocapture. In use, images captured by the video camera may for example beincorporated within a game or interpreted as game control inputs. Inanother embodiment the camera is an infrared camera suitable fordetecting infrared light.

In general, in order for successful data communication to occur with aperipheral device such as a video camera or remote control via one ofthe communication ports of the system unit 500, an appropriate piece ofsoftware such as a device driver should be provided. Device drivertechnology is well-known and will not be described in detail here,except to say that the skilled man will be aware that a device driver orsimilar software interface may be required in the present embodimentdescribed.

FIG. 19 illustrates additional hardware that may be used to processinstructions, in accordance with one embodiment of the presentinvention. Cell processor 528 has an architecture comprising four basiccomponents: external input and output structures comprising a memorycontroller 660 and a dual bus interface controller 670A, B; a mainprocessor referred to as the Power Processing Element 650; eightco-processors referred to as Synergistic Processing Elements (SPEs)610A-H; and a circular data bus connecting the above components referredto as the Element Interconnect Bus 680. The total floating pointperformance of the Cell processor is 218 GFLOPS, compared with the 6.2GFLOPs of the Playstation 2 device's Emotion Engine.

The Power Processing Element (PPE) 650 is based upon a two-waysimultaneous multithreading Power 570 compliant PowerPC core (PPU) 655running with an internal clock of 3.2 GHz. It comprises a 512 kB level 2(L2) cache and a 32 kB level 1 (L1) cache. The PPE 650 is capable ofeight single position operations per clock cycle, translating to 25.6GFLOPs at 3.2 GHz. The primary role of the PPE 650 is to act as acontroller for the Synergistic Processing Elements 610A-H, which handlemost of the computational workload. In operation the PPE 650 maintains ajob queue, scheduling jobs for the Synergistic Processing Elements610A-H and monitoring their progress. Consequently each SynergisticProcessing Element 610A-H runs a kernel whose role is to fetch a job,execute it and synchronized with the PPE 650.

Each Synergistic Processing Element (SPE) 610A-H comprises a respectiveSynergistic Processing Unit (SPU) 620A-H, and a respective Memory FlowController (MFC) 640A-H comprising in turn a respective Dynamic MemoryAccess Controller (DMAC) 642A-H, a respective Memory Management Unit(MMU) 644A-H and a bus interface (not shown). Each SPU 620A-H is a RISCprocessor clocked at 3.2 GHz and comprising 256 kB local RAM 630A-H,expandable in principle to 4 GB. Each SPE gives a theoretical 25.6GFLOPS of single precision performance. An SPU can operate on 4 singleprecision floating point members, 4 32-bit numbers, 8 16-bit integers,or 16 8-bit integers in a single clock cycle. In the same clock cycle itcan also perform a memory operation. The SPU 620A-H does not directlyaccess the system memory XDRAM 526; the 64-bit addresses formed by theSPU 620A-H are passed to the MFC 640A-H which instructs its DMAcontroller 642A-H to access memory via the Element Interconnect Bus 680and the memory controller 660.

The Element Interconnect Bus (EIB) 680 is a logically circularcommunication bus internal to the Cell processor 528 which connects theabove processor elements, namely the PPE 650, the memory controller 660,the dual bus interface 670A,B and the 8 SPEs 610A-H, totaling 12participants. Participants can simultaneously read and write to the busat a rate of 8 bytes per clock cycle. As noted previously, each SPE610A-H comprises a DMAC 642A-H for scheduling longer read or writesequences. The EIB comprises four channels, two each in clockwise andanti-clockwise directions. Consequently for twelve participants, thelongest step-wise data-flow between any two participants is six steps inthe appropriate direction. The theoretical peak instantaneous EIBbandwidth for 12 slots is therefore 96 B per clock, in the event of fullutilization through arbitration between participants. This equates to atheoretical peak bandwidth of 307.2 GB/s (gigabytes per second) at aclock rate of 3.2 GHz.

The memory controller 660 comprises an XDRAM interface 662, developed byRambus Incorporated. The memory controller interfaces with the RambusXDRAM 526 with a theoretical peak bandwidth of 25.6 GB/s.

The dual bus interface 670A,B comprises a Rambus FlexIO® systeminterface 672A,B. The interface is organized into 12 channels each being8 bits wide, with five paths being inbound and seven outbound. Thisprovides a theoretical peak bandwidth of 62.4 GB/s (36.4 GB/s outbound,26 GB/s inbound) between the Cell processor and the I/O Bridge 700 viacontroller 170A and the Reality Simulator graphics unit 200 viacontroller 170B.

Data sent by the Cell processor 528 to the Reality Simulator graphicsunit 530 will typically comprise display lists, being a sequence ofcommands to draw vertices, apply textures to polygons, specify lightingconditions, and so on.

FIG. 20 is an exemplary illustration of scene A through scene E withrespective user A through user E interacting with game clients 702 thatare connected to server processing via the internet, in accordance withan embodiment of the present invention. A game client is a device thatallows users to connect to server applications and processing via theinternet. The game client allows users to access and playback onlineentertainment content such as but not limited to games, movies, musicand photos. Additionally, the game client can provide access to onlinecommunications applications such as VOIP, text chat protocols, andemail.

A user interacts with the game client via controller. In someembodiments the controller is a game client specific controller while inother embodiments, the controller can be a keyboard and mousecombination. In one embodiment, the game client is a standalone devicecapable of outputting audio and video signals to create a multimediaenvironment through a monitor/television and associated audio equipment.For example, the game client can be, but is not limited to a thinclient, an internal PCI-express card, an external PCI-express device, anExpressCard device, an internal, external, or wireless USB device, or aFirewire device, etc. In other embodiments, the game client isintegrated with a television or other multimedia device such as a DVR,Blu-Ray player, DVD player or multi-channel receiver.

Within scene A of FIG. 20, user A interacts with a client applicationdisplayed on a monitor 106 using a controller 100 paired with gameclient 702A. Similarly, within scene B, user B interacts with anotherclient application that is displayed on monitor 106 using a controller100 paired with game client 702B. Scene C illustrates a view from behinduser C as he looks at a monitor displaying a game and buddy list fromthe game client 702C. While FIG. 20 shows a single server processingmodule, in one embodiment, there are multiple server processing modulesthroughout the world. Each server processing module includes sub-modulesfor user session control, sharing/communication logic, usergeo-location, and load balance processing service. Furthermore, a serverprocessing module includes network processing and distributed storage.

When a game client 702 connects to a server processing module, usersession control may be used to authenticate the user. An authenticateduser can have associated virtualized distributed storage and virtualizednetwork processing. Examples items that can be stored as part of auser's virtualized distributed storage include purchased media such as,but not limited to games, videos and music etc. Additionally,distributed storage can be used to save game status for multiple games,customized settings for individual games, and general settings for thegame client. In one embodiment, the user geo-location module of theserver processing is used to determine the geographic location of a userand their respective game client. The user's geographic location can beused by both the sharing/communication logic and the load balanceprocessing service to optimize performance based on geographic locationand processing demands of multiple server processing modules.Virtualizing either or both network processing and network storage wouldallow processing tasks from game clients to be dynamically shifted tounderutilized server processing module(s). Thus, load balancing can beused to minimize latency associated with both recall from storage andwith data transmission between server processing modules and gameclients.

As shown in FIG. 20, the server processing module has instances ofserver application A and server application B. The server processingmodule is able to support multiple server applications as indicated byserver application X1 and server application X2. In one embodiment,server processing is based on cluster computing architecture that allowsmultiple processors within a cluster to process server applications. Inanother embodiment, a different type of multi-computer processing schemeis applied to process the server applications. This allows the serverprocessing to be scaled in order to accommodate a larger number of gameclients executing multiple client applications and corresponding serverapplications. Alternatively, server processing can be scaled toaccommodate increased computing demands necessitated by more demandinggraphics processing or game, video compression, or applicationcomplexity. In one embodiment, the server processing module performs themajority of the processing via the server application. This allowsrelatively expensive components such as graphics processors, RAM, andgeneral processors to be centrally located and reduces to the cost ofthe game client. Processed server application data is sent back to thecorresponding game client via the internet to be displayed on a monitor.

Scene C illustrates an exemplary application that can be executed by thegame client and server processing module. For example, in one embodimentgame client 702C allows user C to create and view a buddy list 720 thatincludes user A, user B, user D and user E. As shown, in scene C, user Cis able to see either real time images or avatars of the respective useron monitor 106C. Server processing executes the respective applicationsof game client 702C and with the respective game clients 702 of users A,user B, user D and user E. Because the server processing is aware of theapplications being executed by game client B, the buddy list for user Acan indicate which game user B is playing. Further still, in oneembodiment, user A can view actual in game video directly from user B.This is enabled by merely sending processed server application data foruser B to game client A in addition to game client B.

In addition to being able to view video from buddies, the communicationapplication can allow real-time communications between buddies. Asapplied to the previous example, this allows user A to provideencouragement or hints while watching real-time video of user B. In oneembodiment two-way real time voice communication is established througha client/server application. In another embodiment, a client/serverapplication enables text chat. In still another embodiment, aclient/server application converts speech to text for display on abuddy's screen.

Scene D and scene E illustrate respective user D and user E interactingwith game consoles 710D and 710E respectively. Each game console 710Dand 710E are connected to the server processing module and illustrate anetwork where the server processing modules coordinates game play forboth game consoles and game clients.

With reference to FIG. 21A, a method 800 for initializing a motioncontroller to determine the size of a TV display (or other type ofdisplay) is described, in accordance with an embodiment of theinvention. At method operation 801, a user holds the controller pointingtoward the TV display. At method operation 802, the TV displays a blackimage or a still image which is used for calibration purposes. In otherembodiments, the state or image displayed by the TV display can be anystate (e.g. on/off, changing over time (e.g. flashing)) or image, or anylevel of image intensity so long as the TV display is identifiable. Atmethod operation 803, an IR projector on the controller projects a beampattern (or patternless) toward the TV display or in a direction wherethe TV display is located. At method operation 804, an IR camera on thecontroller captures an IR image of the beam pattern which is reflectedby the TV display. At method operation 805, the captured IR image isanalyzed, and based on the size/distortion of the IR beam pattern, thedepth Z, yaw and pitch of the controller relative to the TV display aredetermined. At method operation 806, the RGB camera detects the TVdisplay outer frame. At method operation 807, the results of methodoperations 805 and 806 are used to generate an estimation of thephysical size of the TV display.

With reference to FIG. 21B, a method 810 for run-time tracking of the TVdisplay outer frame with the RGB camera is described, in accordance withan embodiment of the invention. At method operation 811, thesize/distortion of the outer frame is analyzed. At method operation 812,the x/y/z position and pitch/yaw of the controller relative to the TVdisplay are generated. At method operation 813, the roll angle isestimated by an accelerometer.

In another embodiment of the invention, instead of using an IR projectorand an IR camera, a laser distance measure is included in thecontroller. With reference to FIG. 22, a method 820 of using a laserdistance measure to calibrate TV display physical size is described, inaccordance with an embodiment of the invention. At method operation 821,a user holds the controller pointing towards the TV display. At methodoperation 822, the controller beams a laser to the TV screen. At methodoperation 823, the distance measurement is determined. At methodoperation 824, an RGB camera detects the TV display outer frame and thelaser dot. At method operation 825, the pitch/roll is determined usingan accelerometer. At method operation 826, it is assumed that the TVdisplay is usually of a rectangular geometry, and based on this, anestimation of the TV display's physical size is generated.

With reference to FIG. 23, a method 830 for providing interaction withan interactive program is described, in accordance with an embodiment ofthe invention. At method operation 831, a beam is projected from acontroller towards a display, or in the general direction of thedisplay. At method operation 832, the size of the projected beam isdetermined by examining a captured image of the projected beam. Atmethod operation 833, a distance from the controller to the display isdetermined based on the determined size of the projected beam. At methodoperation 834, an image of the display is captured at the controller.The display may be configured to generate any type of image/video, ornone at all, so as to facilitate identification of the display in thecaptured image of the display. At method operation 835, the dimensionsof the display are determined based on an analysis of the captured imageof the display and the determined distance to the display. At methodoperation 836, successive images of the display are captured at thecontroller. These successive images of the display are analyzed so as todetermine position/orientation/movement of the controller. At methodoperation 837, the determined position/orientation/movement of thecontroller is provided as input to an interactive program so as to causean action in the interactive program.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

1. In a computer-based system having a display and a controller devicefor interfacing with an interactive program, a method for determiningthe position of the controller device, the method comprising: (a)initializing the computer-based system to establish a starting positionof the controller device relative to the display, the initializingincluding, (i) capturing an image of the display using a cameraintegrated in the controller device; (ii) capturing an image of aradiation pattern projected by the controller device; and (iii)analyzing the captured image of the display and the image of theradiation pattern projected by the controller device to determine a sizeof the display in the captured image at a distance of the controllerdevice as determined by the image of the projected radiation pattern;(b) capturing successive images of the display at the controller device;(c) determining a position of the controller device relative to thedisplay based on a perspective distortion of the display in the capturedsuccessive images of the display; (d) providing the determined positionof the controller to the computer-based system to interface with theinteractive program to cause an action by the interactive program. 2.The method of claim 1, further comprising, determining an orientation ofthe controller device based on a location and orientation of the displayin the captured image of the display.
 3. The method of claim 2, whereinthe orientation of the controller device is selected from the groupconsisting of pitch, roll, and yaw.
 4. The method of claim 1, furthercomprising, determining movement of the controller device by trackingthe position of the controller at the time of each captured successiveimage.
 5. The method of claim 1, wherein the radiation pattern projectedby the controller is generated by an IR projector.
 6. The method ofclaim 1, further comprising, when the position of the controller devicecannot be determined based on the captured successive images of thedisplay, then performing the following operations: projecting theradiation pattern from the controller device; capturing successiveimages of the radiation pattern at the controller device; and estimatingthe position of the controller device based on the captured successiveimages of the radiation pattern.
 7. The method of claim 1, wherein theoperation of capturing an image of the display includes capturing animage of a portion of the display sufficient to enable determination ofthe size of the display.
 8. The method of claim 1, wherein the operationof capturing successive images of the display includes capturingsuccessive images of a portion of the display sufficient to enabledetermination of the position of the controller device.
 9. In acomputer-based system having a display and a controller device forinterfacing with an interactive program, a method for determining theposition of the controller device, the method comprising: (a) receivingdimensions of the display input by a user of the computer-based system;(b) capturing successive images of the display at the controller device;(c) determining a position of the controller device relative to thedisplay based on the dimensions of the display and a perspectivedistortion of the display in the captured successive images of thedisplay; (d) providing the determined position of the controller to thecomputer-based system to interface with the interactive program to causean action by the interactive program.
 10. The method of claim 9, whereinthe dimensions of the display include an aspect ratio of the display anda diagonal measurement of the display.
 11. A system for interfacing withan interactive program, comprising: a display for displaying theinteractive program; a controller device for interfacing with theinteractive program, the controller device including, a projector forprojecting a radiation pattern, a first camera for capturing an image ofthe projected radiation pattern, a second camera for capturing images ofthe display; initialization logic for initializing the system toestablish a starting position of the controller device relative to thedisplay, the initialization logic configured to, (i) capture an image ofthe display using the second camera, (ii) capture an image of aradiation pattern projected by the projector using the first camera, and(iii) analyzing the captured image of the display and the image of theradiation pattern to determine a size of the display in the capturedimage at a distance of the controller device as determined by the imageof the projected radiation pattern; position determination logic fordetermining the position of the controller device relative to thedisplay, the position determination logic configured to, (i) capturesuccessive images of the display at the controller device using thesecond camera, (ii) determine a position of the controller devicerelative to the display based on a perspective distortion of the displayin the captured successive images of the display; communication logicfor providing the determined position of the controller to thecomputer-based system to interface with the interactive program.
 12. Thesystem of claim 11, further comprising, orientation determination logicfor determining an orientation of the controller device based on alocation and orientation of the display in the captured successiveimages of the display.
 13. The system of claim 12, wherein theorientation of the controller device is selected from the groupconsisting of pitch, roll, and yaw.
 14. The system of claim 11, furthercomprising, movement determination logic for determining movement of thecontroller device by tracking the position of the controller at the timeof each captured successive image.
 15. The system of claim 11, furthercomprising, auxiliary position determination logic configured to performthe following operations when the position of the controller devicecannot be determined based on the captured successive images of thedisplay: projecting the radiation pattern from the controller device;capturing successive images of the radiation pattern at the controllerdevice; and estimating the position of the controller device based onthe captured successive images of the radiation pattern.
 15. The systemof claim 11, wherein the capture of an image of the display includescapture of an image of a portion of the display sufficient to enabledetermination of the size of the display.
 16. The system of claim 11,wherein the capture of successive images of the display includes captureof successive images of a portion of the display sufficient to enabledetermination of the position of the controller device.
 17. In acomputer-based system having a display and a controller device forinterfacing with an interactive program, means for determining aposition of the controller device, the means for determining positioncomprising: (a) means for initializing the computer-based system toestablish a starting position of the controller device relative to thedisplay, the means for initializing including, (i) means for capturingan image of the display using a camera integrated in the controllerdevice; (ii) means for capturing an image of a radiation patternprojected by the controller device; and (iii) means for analyzing thecaptured image of the display and the image of the radiation patternprojected by the controller device to determine a size of the display inthe captured image at a distance of the controller device as determinedby the image of the projected radiation pattern; (b) means for capturingsuccessive images of the display at the controller device; (c) means fordetermining a position of the controller device relative to the displaybased on a perspective distortion of the display in the capturedsuccessive images of the display; (d) means for providing the determinedposition of the controller to the computer-based system to interfacewith the interactive program to cause an action by the interactiveprogram.
 18. The means for determining position of claim 17, furthercomprising, means for determining an orientation of the controllerdevice based on a location and orientation of the display in thecaptured image of the display.
 19. The means for determining position ofclaim 18, wherein the orientation of the controller device is selectedfrom the group consisting of pitch, roll, and yaw.
 20. The means fordetermining position of claim 17, further comprising, means fordetermining movement of the controller device by tracking the positionof the controller at the time of each captured successive image.
 21. Themeans for determining position of claim 17, wherein the radiationpattern projected by the controller is generated by an IR projector. 22.The means for determining position of claim 17, further comprising,means for auxiliary position determination, the means for auxiliaryposition determination configured to perform the following operationswhen the position of the controller device cannot be determined based onthe captured successive images of the display: projecting the radiationpattern from the controller device; capturing successive images of theradiation pattern at the controller device; and estimating the positionof the controller device based on the captured successive images of theradiation pattern.
 23. The means for determining position of claim 17,wherein the capturing an image of the display includes capturing animage of a portion of the display sufficient to enable determination ofthe size of the display.
 24. The means for determining position of claim17, wherein the capturing successive images of the display includescapturing successive images of a portion of the display sufficient toenable determination of the position of the controller device.
 25. Acomputer program product comprising instructions embodied on a computerreadable medium, the computer program product for determining theposition of a controller device for interfacing with an interactiveprogram, the computer program product including: (a) instructions forreceiving dimensions of a display input by a user; (b) instructions forcapturing successive images of the display at the controller device; (c)instructions for determining a position of the controller devicerelative to the display based on the dimensions of the display and aperspective distortion of the display in the captured successive imagesof the display; and (d) instructions for providing the determinedposition of the controller to the computer-based system to interfacewith the interactive program to cause an action by the interactiveprogram.
 26. The computer program product of claim 25, wherein thedimensions of the display include an aspect ratio of the display and adiagonal measurement of the display.
 27. The computer program product ofclaim 25, further including, instructions for determining an orientationof the controller device based on a location and orientation of thedisplay in the captured image of the display.
 28. The computer programproduct of claim 27, wherein the orientation of the controller device isselected from the group consisting of pitch, roll, and yaw.
 29. Thecomputer program product of claim 25, further comprising, instructionsfor determining movement of the controller device by tracking theposition of the controller at the time of each captured successiveimage.